Papers by Lucas Torroba Hennigen
A Measure-Theoretic Characterization of Tight Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Language modeling is a core task in natural language processing. |
| Approach: | They propose to characterize leakage onto the set of infinite sequences by a measure-theoretic approach. |
| Outcome: | The proposed language model families are tight, meaning they will not leak . the proposed language models are based on the 'sequence leakage' hypothesis . |
Deriving Language Models from Masked Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | Masked language models do not define an explicit distribution over language, but they assume that tokens masked out are conditionally independent given the unmasked tokens. |
| Approach: | They propose to use a set of MLM's unary conditionals to construct a fully-connected Markov random field over the input to deduce an explicit joint distribution from MLMs. |
| Outcome: | The proposed method outperforms existing Markov random field-based approaches and outperformed the original model's conditionals. |
An Ordinal Latent Variable Model of Conflict Intensity (2023.acl-long)
Copied to clipboard
| Challenge: | Advances in automated event extraction yield massive data sets of “who did what to whom” micro-records that enable data-driven approaches to monitoring conflict. |
| Approach: | They propose a probabilistic generative model that assumes each observed event is associated with a latent intensity class. |
| Outcome: | The proposed model obtains comparatively good held-out predictive performance on a conflictual to cooperative scale. |
Machine Reading of Historical Events (2020.acl-main)
Copied to clipboard
| Challenge: | Using a short text description of an event, we can extract relevant sentences from Wikipedia and apply a combination of task-specific and general-purpose feature embeddings for the classification. |
| Approach: | They propose to use Wikipedia sentences to extract relevant sentences and apply feature embeddings to the task. |
| Outcome: | The proposed model outperforms the historical event ordering task and the event focus time task in the literature. |
Probing as Quantifying Inductive Bias (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained contextual representations have led to performance improvements on downstream tasks. |
| Approach: | They propose a Bayesian framework that quantifies the amount of inductive bias that the representations encode on a specific task. |
| Outcome: | The proposed framework alleviates many problems found in probing and can offer better inductive bias than BERT. |
Classifying Dyads for Militarized Conflict Analysis (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing research examines the origins of militarized conflict by examining bi-lateral relationships between entity pairs and multi-lateral relations among multiple entities. |
| Approach: | They propose to use Wikipedia to model dyadic and systemic causes to compare their correlations with conflict between two entities. |
| Outcome: | The proposed graphs show that Wikipedia articles of allies are semantically more similar than enemies. |
Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing studies show that multilingual pre-trained models can learn to generalise across languages . however, it remains unclear how these models learn to learn multilingual representations . |
| Approach: | They propose a hypothesis that multilingual pre-trained models can derive language-universal abstractions about grammar by aligning morphosyntactic markers that fulfil a similar grammatical function across languages. |
| Outcome: | The proposed model can derive language-universal abstractions even without explicit supervision. |
Intrinsic Probing through Dimension Selection (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on probing for linguistic structure in word embeddings has focused on intrinsic probing, but what these representations encode about linguistic structures remains unclear. |
| Approach: | They propose a framework that allows us to determine whether linguistic information in word embeddings is dispersed or focal. |
| Outcome: | The proposed framework allows us to determine whether linguistic information in word embeddings is dispersed or focal. |
Generalizing Backpropagation for Gradient-Based Interpretability (2023.acl-long)
Copied to clipboard
| Challenge: | Several feature-attribution methods for interpreting deep neural networks rely on computing the gradients of a model’s output with respect to its inputs, but they reveal little about the inner workings of the model itself. |
| Approach: | They propose a generalized backpropagation algorithm that generalizes the gradient computation of a model to efficiently compute other interpretable statistics about the gradient graph of neural networks. |
| Outcome: | The proposed generalized algorithm can be used to compute other interpretable statistics about the gradient graph of a neural network, such as the highest-weighted path and entropy. |